Clustering Template Based Web Documents

More and more documents on theWorld WideWeb are based on templates. On a technical level this causes those documents to have a quite similar source code and DOM tree structure. Grouping together documents which are based on the same template is an important task for applications that analyse the template structure and need clean training data. This paper develops and compares several distance measures for clustering web documents according to their underlying templates. Combining those distance measures with different approaches for clustering, we show which combination of methods leads to the desired result. As more and more documents on the World Wide Web are generated automatically by Content Management Systems (CMS), more and more of them are based on templates. Templates can be seen as framework documents which are filled with different contents to compile the final documents. They are a standard (if not even essential) CMS technology. Templates provide the managed web sites with an easy to manage uniform look and feel. A technical side effect is that the source code of template generated documents is always very similar. Several algorithms have been developed to automatically detect these template structures in order to identify and / or extract particular parts of a document such as the main content. These structure detection algorithms depend on training sets of documents which are all supposed to be based on the same template. Only few works though address the problem of actually creating these clean training sets or verifying that the documents in a given training set are all based on the same template. Approaches trying to handle this problem usually involve clustering the documents to group together those which have large structural similarities. However, to our knowledge this process has never been analysed or verified itself. In this paper we take a closer look at web document distance measures which are supposed to reflect template related structural similarities and dissimilarities. We will evaluate the distance measures both under the aspect of computational costs and – given different clustering approaches – how suitable they are to cluster documents according to their underlying templates. The evaluation is based on a corpus of 500 web documents, taken from different sub-categories of five different web sites. We proceed as follows: In section 1 we give an overview over related works in this fields, focussing in particular on distance measures for web documents which take into account mainly structural information. In 2 and 3 we describe six different distance measures in more detail and some standard cluster analysis algorithms we used. The experiment setup and results are presented in section 4 before we conclude the paper in 5 with a discussion of the results.